docs: document speculation profiles and how to produce them - #2316

The first version of _speculation_profile_metadata() read K off --draft_length
unconditionally and derived max_supported_k as block_size - 1. Both are wrong for
DFlash, which is the method this profile is most needed for.

Reading the engine wrappers: DFLASH is configured by --block_size, which both
models/vllm.py and models/sglang.py forward as num_speculative_tokens /
speculative_num_draft_tokens while ignoring --draft_length -- sglang.py emits an
explicit warning saying so. Every other method uses --draft_length as
speculative_num_steps. Labelling the vectors with K from the wrong flag would be
silent and plausible, so derive it per method.

max_supported_k now defaults to the measured K rather than block_size - 1.
--block_size here is the number handed to the engine as num_speculative_tokens,
which despite the shared name is not the trained dflash_block_size in the
checkpoint config. specdec_bench cannot observe the real architectural ceiling,
and publishing an unverifiable one is worse than publishing none.

Signed-off-by: Ye Yu <yeyu@nvidia.com>

Three points from CodeRabbit on NVIDIA#2247.

Publish identifiers, not paths. The profile is intended to ship alongside a
checkpoint, so serialising args.model_dir / args.draft_model_dir verbatim would
bake internal cluster layout (/lustre/fsw/portfolios/...) into a public artifact,
and an absolute path is not portable for a reader in any case. checkpoint_id()
reduces a path to its trailing org/model, which is both the useful part and the
HuggingFace-style id. configuration.json still records full paths for local
debugging.

Clear profile metadata when a run has no --save_dir. The metadata is class-level
state (following the existing Metric.update_directory pattern), so an in-process
second run -- the AR-vs-K sweep this schema is built for is exactly that shape --
could otherwise inherit the previous run's destination.

Declare __all__. Not re-exported from specdec_bench/__init__.py as suggested:
that module deliberately exposes only __version__ and must stay importable
without modelopt (the vLLM container has no modelopt), so widening it would break
its own convention. Noted inline so the omission reads as deliberate.

Signed-off-by: Ye Yu <yeyu@nvidia.com>

The first real measurement (nvidia/MiniMax-M2.7-DFlash on MT-Bench, 30653 decode
steps) failed the profile's own consistency check: 1 + sum(marginals) = 2.4733
against a reported 2.5467. The vectors were right; the mean was the wrong one.

Average_AL averages per-request accept length over requests, weighting a short
request the same as a long one. The acceptance vectors describe a per-*step*
distribution -- both dynamo's mocker and vLLM's synthetic sampler draw a length
per decode step -- so the identity was comparing incompatible quantities and would
have flagged every real run.

mean_accept_length is now computed from the acceptance-length histogram, which is
what the vectors describe. The per-request figure is kept as
mean_accept_length_per_request, since published model cards do not always state
which mean they quote and the comparison is worth preserving.

This also sharpens what the check guards. Both sides now derive from the same
histogram, so the identity holds exactly whenever the published vector spans every
observed acceptance length -- meaning what it actually detects is truncation: a
num_speculative_tokens that understates the K the run used cuts the vector short
and would otherwise silently describe a weaker draft than was measured. Given K is
derived from CLI flags whose meaning varies by method, that is the failure mode
worth catching. Test updated accordingly, plus one pinning both means on the real
MiniMax histogram.

Signed-off-by: Ye Yu <yeyu@nvidia.com>

A measured acceptance profile is only useful if it reaches the deployment. Today
it stops at the benchmark output directory, so consumers guess instead -- dynamo's
simulator models every draft model in existence with one hardcoded acceptance
vector. This carries the measurement into the export, next to the weights.

export_speculative_decoding() gains an optional speculation_profile path, plumbed
through scripts/export_hf_checkpoint.py as --speculation_profile.

The exporter deliberately only *transports* a profile; it does not build one.
Acceptance is measured by a benchmark harness that commonly runs in an engine
container without modelopt installed -- the MiniMax-M2.7 DFlash measurement ran
under vllm/vllm-openai:nightly, where configuration.json recorded
modelopt_version: null. The producer therefore cannot import from this side, so
this side stays a carrier: it validates the file is JSON carrying a
schema_version, and copies it in.

For the same reason the version is recorded rather than checked against a
constant. Producers own the schema; pinning an expected version here would create
a second source of truth that drifts.

With no profile supplied an unmeasured stub is written, so consumers can tell "not
measured" from "predates the schema" -- absent then means a genuinely old
checkpoint rather than an ambiguous one.

Hooked in export_speculative_decoding() rather than inside each exporter's
export(): one call site covers Eagle, EagleMedusa, DFlash, Domino and DSpark, so a
newly added method cannot silently ship without a profile.

Verified by round-tripping the real measured profile for
nvidia/MiniMax-M2.7-DFlash (conditional [0.816, 0.777, 0.750], AL 2.925) through
the exporter byte-identically.

Signed-off-by: Ye Yu <yeyu@nvidia.com>

…-export

ar_validate.py already measures acceptance position-by-position -- validate_online
breaks on first rejection, so it walks exactly the longest-prefix distribution --
then collapses it into a scalar and prints it. Nothing downstream can consume
that: not CI regression gating, not the export step, not a deployment.

validate_online now also returns the per-step acceptance-length histogram, and
--output_json writes the same speculation_profile.json schema specdec_bench
produces.

Two producers, one schema, for different moments: this one runs inside the
training loop with no serving engine, so acceptance can be tracked as a
checkpoint trains; specdec_bench measures the deployed engine. A consumer should
not have to care which produced a profile. Verified on the real MiniMax-M2.7
DFlash histogram -- both emit byte-identical conditional
[0.816082, 0.776577, 0.749591], marginal [0.816082, 0.633751, 0.475054] and
mean_accept_length 2.924887.

The conversion is reimplemented rather than imported, deliberately. specdec_bench's
copy must stay importable without modelopt because it runs in engine containers
where modelopt is absent -- the MiniMax measurement recorded
modelopt_version: null -- and importing into modelopt from examples/ is not
possible either. The shared piece is small and now pinned by tests on both sides;
a third producer would be the point to extract it properly.

validate_online's return arity changes from 2 to 3. It is not re-exported from any
__init__, so it is not public API, and all three in-repo call sites are updated.

--output_json is written before the --ar_lower_bound check: an out-of-bounds AR is
still worth having on disk, and raising first would discard the measurement that
explains the failure.

Signed-off-by: Ye Yu <yeyu@nvidia.com>

…tion-profile-docs

Completes the speculation-profile work: the artifact, the two producers, the
export step, and the traps that make a wrong profile look right.

The framing throughout is why it exists rather than what the fields are called: a
draft checkpoint's weights say nothing about how good it is, so deployment tooling
guesses -- dynamo's simulator models every draft model in existence with one
hardcoded acceptance vector, and a strong draft and a weak one produce the same
capacity estimate.

Three things are documented specifically because getting them wrong is silent:

- Both acceptance conventions are published and labelled. dynamo's mocker wants
  conditional rates; vLLM's synthetic rejection sampler wants marginal ones.
  Emitting one and letting a consumer assume the other is plausible-looking and
  wrong.
- mean_accept_length is per *step*, not per request, and satisfies
  AL = 1 + sum(marginals). The per-request mean is reported separately; the two
  differ on real data.
- accept_length_model says whether K may be extrapolated -- chain-drafted methods
  truncate cleanly, block-parallel ones (DFlash, DSpark) must be measured per K.

Also records the generation-length lesson from validating against a published
card, since it cost a full GPU run: measuring nvidia/MiniMax-M2.7-DFlash at 512
tokens gives AL 2.47 against that card's 3.05, while the card's stated 4096 gives
2.92, within 4.1%. Truncation removes the long predictable stretches where drafts
do best. Match the published setup before economising anywhere.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Ye Yu <yeyu@nvidia.com>

…-export

…-ar-validate

…profile-docs

…tion-profile-docs

…-export

Adds the case the existing rejection tests miss. Those cover valid JSON of the
wrong shape; this one never parses -- a half-written profile from an interrupted
run is the realistic way to produce it, and it must fail on the parser rather than
slip through.

Signed-off-by: Ye Yu <yeyu@nvidia.com>

…-ar-validate

If every sample failed, or osl was too small to produce a single decode step, the
histogram is empty. Writing measured=true then advertises a draft that accepts
nothing, which reads identically to a genuinely terrible draft. Warn and skip
instead.

Signed-off-by: Ye Yu <yeyu@nvidia.com>

…profile-docs

…tion-profile-docs

…profile-docs

Signed-off-by: Ye Yu <yeyu@nvidia.com>

# Conflicts:
#	tests/unit/torch/speculative/plugins/test_hf_dflash.py

Signed-off-by: Ye Yu <yeyu@nvidia.com>

# Conflicts:
#	examples/speculative_decoding/scripts/export_hf_checkpoint.py

…hapes

The docstring claimed build_profile requires Conditional_Acceptance_Rate. It
never reads it -- conditionals are recomputed from the dense marginals -- so a
second producer of this schema would have synthesized that key for nothing.
Now lists what is consumed: Joint_Acceptance_Rate, Acceptance_Length_Histogram
and Average_AL as the fallback.

per_category was documented as {category: {mean_accept_length, ...}} but the
only caller passes out["Category_AL"], which is {category: float} in both
producers. That value is serialized verbatim into the published schema, so the
docstring described a shape consumers would never find in the file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Ye Yu <yeyu@nvidia.com>
(cherry picked from commit 0895900)

…sistency

- A bad --speculation_profile used to fail *after* exporter.export() had
  written the checkpoint, leaving a partial export behind (and a stale
  speculation_profile.json able to survive overwritten weights). Split the
  read/validate step out of write_speculation_profile as
  load_speculation_profile and call it before export, so an invalid profile
  fails while the destination is still empty. The validated record is
  handed to the write step rather than re-read.

- mean_consistency failed on a legitimately unmeasured profile: an empty
  histogram gives mean_accept_length 0.0 while the all-zero marginal
  implies 1.0, so the check reported a discrepancy for a profile that never
  claimed a measurement. It is now None -- not applicable rather than
  failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Ye Yu <yeyu@nvidia.com>
(cherry picked from commit 62de9cb)

…ting

Chenhan on NVIDIA#2313: "the order does not seem right."

The ordering is causal -- acceptance can only be measured against a servable
checkpoint, so the measurement follows export no matter how this is arranged.
What was avoidable is the re-export used to deliver the result.

Attaching a measured profile is pure file I/O: read, validate, write one JSON.
write_speculation_profile touched exporter state only in the unmeasured-stub
branch, which needs the model to name its speculation method. So the supplied
path is now a module-level read_speculation_profile(), and a small script uses
it to attach a profile to an existing export:

    export_hf_checkpoint.py --export_path ckpt/
    specdec_bench/run.py    --save_dir  run/
    attach_speculation_profile.py --export_path ckpt/ \
        --speculation_profile run/speculation_profile.json

instead of reloading multi-GB weights and rewriting every shard to place one
small file beside them. The profile is validated before it replaces an existing
one, and the previous file is backed up, so a malformed profile cannot leave a
checkpoint worse off than it found it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Ye Yu <yeyu@nvidia.com>
(cherry picked from commit 0fc08ac)

…ng accuracy

- Re-export speculation_profile from specdec_bench/__init__.py, per the
  package's declared convention. Safe here where the modelopt import above
  is not: the module is stdlib-only by design (math), which is what lets the
  benchmark run in a vLLM container without modelopt.

- build_profile's docstring claimed it requires Conditional_Acceptance_Rate.
  It never reads it -- conditionals are recomputed from the dense marginals.
  A second producer of this schema would have synthesized that key for
  nothing. Now lists what is actually consumed: Joint_Acceptance_Rate,
  Acceptance_Length_Histogram, and Average_AL as the fallback.

- per_category was documented as {category: {mean_accept_length, ...}} but
  the only caller passes out["Category_AL"], which is {category: float} in
  both producers. That value is serialized verbatim into the published
  schema, so the docstring described a shape consumers would never find.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Ye Yu <yeyu@nvidia.com>
(cherry picked from commit 35a6094)

…ontract

- validate_online widened its return from a 2-tuple to a 3-tuple with no
  deprecation path. AcceptanceRateValidation is public (subclassed by
  HFARValidation and MegatronARValidation), so `ids, ar = validate_online(...)`
  broke, and it left the method inconsistent in arity with its sibling
  validate(). Back to a 2-tuple, with the histogram exposed as
  last_length_histogram and a class-level default so reading it before a call
  is defined.

- The exporter's unmeasured stub was a third, incompatible shape: no
  num_speculative_tokens, no rate vectors, no accept_length_model, and
  schema_version None. That branch runs on every export without
  --speculation_profile, so it is the most-shipped version of this artifact --
  a consumer written against the documented schema KeyError'd instead of
  reading measured:false. schema_version None also meant "declares no schema",
  the precise ambiguity the stub exists to remove. Now matches stub_profile()
  field for field.

- ar_validate emitted neither method nor accept_length_model, so a consumer
  could not tell whether K may be extrapolated -- and assuming it can for a
  measured_per_k draft silently scales a profile valid at one K only. Both
  threaded through, reusing build_profile's defaulting rule.

- validation reported mean_consistency and marginal_monotonicity over rate
  vectors the profile withholds under block verification. Both now null there.

- marginal_monotonicity was computed and serialized but never surfaced, so a
  malformed histogram stayed silent. Now warned on alongside mean_consistency.

- The histogram/AR test asserted arithmetic on a hand-built dict and never
  called validate_online, so it did not guard the mismatch its docstring
  described. It now drives validate_online and checks the returned AR against
  the histogram that call produced.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Ye Yu <yeyu@nvidia.com>